Adaptive metadata refreshing

ABSTRACT

Techniques are described for managing the optimized refreshing of metadata associated with online and live systems. In some implementations, a set of metadata modules associated with one or more entities are identified, the metadata modules defining metadata associated with a particular data model for the associated entities. A request to initiate a refreshing of the metadata for a subset of the set of metadata modules is identified. Each metadata module from the subset of the set of metadata modules is prioritized into a prioritization order. A determination is made as to whether two or more idle database connections are available. In response to determining that two or more idle database connections are available, a concurrent refresh of the subset of the set of metadata modules is initialized in the prioritization order.

BACKGROUND

The present disclosure relates to a system and computerized method formanaging the optimized refreshing of metadata associated with online andlive systems.

Many systems have metadata, especially for Web Service APIs (applicationprogramming interfaces). The backend data model structure for suchsystems may change from time to time or on a regular basis. In order forlive applications to access the available data correctly, the metadataassociated with the systems must be updated in a timely fashion to allowusers and applications to correctly access and interact with thesystem's data structure and data values.

SUMMARY

Implementations of the present disclosure are generally directed tomanaging the optimized refreshing of metadata associated with online andlive systems. In one example implementation, a computerized methodexecuted by hardware processors can be performed. The example method cancomprise identifying a set of metadata modules associated with one ormore entities, the metadata modules defining metadata associated with aparticular data model for the associated entities. A request to initiatea refreshing of the metadata for a subset of the set of metadata modulesis identified. Each metadata module from the subset of the set ofmetadata modules is prioritized into a prioritization order. Adetermination is made as to whether two or more idle databaseconnections are available. In response to determining that two or moreidle database connections are available, a concurrent refresh of thesubset of the set of metadata modules is initialized in theprioritization order.

Implementations can optionally include one or more of the followingfeatures. In some instances, in response to determining that only oneidle database connection is available, a single thread refresh of thesubset of the set of metadata modules is initiated. In those instances,the single thread refresh of the set of metadata modules comprises asequential processing of metadata modules in the subset of the set ofmetadata modules.

In some instances, prior to determining whether two or more idledatabase connections are available, the method includes determiningwhether a set of system resource metrics satisfy a concurrent refreshthreshold, and, in response to determining the set of system resourcemetrics do not satisfy the concurrent refresh threshold, a single threadrefresh of the subset of the set of metadata modules is initiated. Inthose instances, the concurrent refresh threshold may include at leastone of a particular current CPU usage amount threshold and a currentavailable memory threshold, wherein the current available memorythreshold represents an available memory greater than the requiredmemory associated with the concurrent refresh.

In some instances, initiating the concurrent refresh of the subset ofthe set of metadata modules comprises determining a CPU core number, C,of a particular set of servers to perform the refresh and determiningthe number of idle database connections, #CONN, at the particular set ofservers. In response to determining that C is less than or equal to#CONN, a concurrent refresh of the subset of the set of metadata modulesusing #CONN threads at the particular set of servers is performed, whilein response to determining that C is greater than #CONN, a concurrentrefresh of the subset of the set of metadata modules using C+1 threadsat the particular set of servers is performed.

In some instances, the request to initiate a refreshing of the metadatafor a subset of the set of metadata modules comprises one of a manualrequest by a particular user to initiate the refreshing of the metadata,an automatic request to initiate the refreshing of the metadata inresponse to the expiration of a cache associated with the metadatamodules, an automatic request to initiate the refreshing of the metadatain response to a determination that at least one entity associated withat least one of the metadata modules has been updated, and an automaticrequest to initiate the refreshing of the metadata in response to adetermination that at least one new entity is newly added.

In some instances, prioritizing each metadata module from the subset ofthe set of metadata modules into a prioritization order comprisesprioritizing the metadata modules in a descending order based on anaverage refresh time for each of the metadata modules. In thoseinstances, at least one metadata module in the subset of the set ofmetadata modules is associated with a relatively higher priority, wherethe relatively higher priority is distinct from the average refresh timeof the at least one metadata module. In such instances, the at least onemetadata module is provided a relatively higher priority in theprioritization order than metadata modules having a relatively longeraverage refresh time than the at least one metadata module. In someinstances, the average refresh time for each metadata module iscalculated based on metrics calculated for a predetermined number ofprior refresh times for each metadata module.

Similar operations and processes may be performed in a system comprisingat least one process and a memory communicatively coupled to the atleast one processor where the memory stores instructions that whenexecuted cause the at least one processor to perform the operations.Further, a non-transitory computer-readable medium storing instructionswhich, when executed, cause at least one processor to perform theoperations may also be contemplated. In other words, while generallydescribed as computer implemented software embodied on tangible,non-transitory media that processes and transforms the respective data,some or all of the aspects may be computer implemented methods orfurther included in respective systems or other devices for performingthis described functionality. The details of these and other aspects andembodiments of the present disclosure are set forth in the accompanyingdrawings and the description below. Other features, objects, andadvantages of the disclosure will be apparent from the description anddrawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an example system forimplementing an optimized concurrent metadata refresh process.

FIG. 2 represents an example flow for concurrently refreshing metadataassociated with a plurality of modules.

FIG. 3 is an example illustration of a workflow for concurrent threadmetadata refresh where the idle number of database connections, #conn,is greater than the number of CPU cores, C, in one or more systemsperforming the refreshing.

FIG. 4 is an example illustration of a workflow for concurrent threadmetadata refresh where #conn is less than or equal to C in the one ormore systems performing the refreshing.

DETAILED DESCRIPTION

The present disclosure describes systems and methods for managing theoptimized refreshing of metadata associated with online and livesystems, such as those associated with a Web Service API, as well as anycomputer system or application that is associated with underlyingmetadata for accessing backend objects. As described, the backend datamodel structure for a database or data set associated with the WebService API or other systems can be updated or modified byadministrators or users at various times. Fields or properties can beadded, removed, and/or modified—in such instances, associated APIs orother components attempting to access the system must know theparticulars of the backend data model structure in order to fully accessand interact with the data and its particulars. For example, if a fieldor property is added for an entity and the metadata is not refreshed,future queries to the entity may not return the new field or propertyuntil the metadata is updated, providing users with incomplete sets ofdata. If a field or property is deleted for a particular entity and themetadata is not refreshed, queries to the entity may provide erroneousor redundant data for the now deleted field or property. The problem canbe further exasperated when an insert or update operation is performed.Without refreshing the metadata, data for deleted fields or propertiescan be provided erroneously, while no data associated with newly addedfields or properties may be returned, thereby leading to dirty andincorrect data within the system.

When entire entities are added or deleted, further problems may be seen.For example, if a newly added entity is added to the data modelstructure, existing queries based on an outdated set of metadata willnot return the newly added entity. Conversely, where an entity isremoved or deleted, customers or applications may obtain responsesincluding data from the new deleted entities where no data should bereturned.

Refreshing metadata is an important and necessary task to be performedon a regular interval, as well as in response to system events and/oruser triggering. In many current systems, hundreds or thousands ofentities may exist in systems. In current systems, updating metadataentities or modules is performed sequentially in a one-by-one process,where the time required to perform the entirety of the metadata updateequals the full sum of the time to refresh each individual entity'smetadata. While individual entities may only take a very short time torefresh, the refreshing of a large number of entities can cause systemslowdowns and resource depletion. Considered along with the networktransmission time, end-to-end scenarios for metadata refreshing cancause inconvenient system performance and timing for users and theirapplications/systems.

The present disclosure describes a dynamic method for performingmetadata refresh, wherein the type of refresh (e.g., concurrent orsingle thread) can be determined based on the current status andsituation of the servers available to perform the refresh (e.g., cloudand/or on-premise servers), as well as the details related to theimplementation of the concurrent refresh, where available (e.g., numberof threads available to perform the refresh). Using such a dynamicsolution, the performance and timing of the metadata refresh can begreatly improved.

In the present solution, two types of refresh are available and areselected dynamically at refresh-time: a single thread refresh (i.e.,using a single thread to process each metadata refresh) and a concurrentrefresh (i.e., which uses multiple threads to process the metadata ofvarious entities and/or metadata modules concurrently). In someinstances, multiple system metrics and available resources aredynamically considered at the time of the refresh. Such system metricscan include, for example, a current CPU usage, a current memory usage,and a number of available database connections. Other dynamicconsiderations may also be brought into the evaluation, including user-or administrator-specific concurrent refresh rules and parameters.

In general, the solutions provided herein can provide up to a 200%improvement in refresh performance. Currently, the time of a singlethread refresh is the sum of each refresh time T_(i), or T₁+T₂+. . .T_(n). In contrast, the concurrent thread refresh time using the presentsolution is ideally the maximum of{T_(i)T_(2m−i+1)+T_(2m+i)+T_(4m−i+1)+. . . }, i=1, 2, . . . m. However,in some extreme situation, T₁ may be greater than the sum of all otherrefresh times such that the execution time is T₁.

Turning to the illustrated implementation, FIG. 1 is a block diagramillustrating an example system 100 for implementing an optimizedconcurrent metadata refresh process. As illustrated in FIG. 1, system100 is associated with a cloud-based system capable of sharing andcommunicating information across devices and systems (e.g., cloud system102 or individual cloud servers 104, external data sources 160, andclient 180, among others, via network 150). In some implementations,only a single cloud server 104 may available in the cloud system 102,while in others multiple cloud servers 104, 138 may be available andused for the metadata refresh process. In some instances, while multipleservers 104, 138 may be included in the system, only a subset of theprocessors 108, 140 (or portions of their respective CPU cores)associated with those servers 104, 138 may be available to perform therefresh processing. In some instances, non-cloud-based systems, such ason-premise systems, may use or adapt the processes described herein.Although components are shown individually, in some implementations,functionality of two or more components, systems, or servers may beprovided by a single component, system, or server.

As used in the present disclosure, the term “computer” is intended toencompass any suitable processing device. For example, cloud server 104,external data source 160, and/or client 180 may be any computer orprocessing device such as, for example, a blade server, general-purposepersonal computer (PC), Mac®, workstation, UNIX-based workstation, orany other suitable device. Moreover, although FIG. 1 illustrates aplurality of cloud servers 104, the cloud system 102 can be implementedusing a single system or more than those illustrated, as well ascomputers other than servers, including a server pool. In other words,the present disclosure contemplates computers other than general purposecomputers, as well as computers without conventional operating systems.Similarly, the client 180 may be any system which can request dataand/or interact with the cloud system 102 and external data sources 160.The client device 180, in some instances, may be a desktop system, andclient terminal, or any other suitable device, including a mobiledevice, such as a smartphone, tablet, smartwatch, or any other mobilecomputing device. In general, each illustrated component may be adaptedto execute any suitable operating system, including Linux, UNIX,Windows, Mac OS®, Java™, Android™, Windows Phone OS, or iOS™, amongothers.

The cloud system 102 may be associated with the management andoperations of one or more applications and/or application programminginterfaces (APIs) for receiving queries associated with one or more datasources, including external data sources 160. The cloud system 102 maymanage the execution or interpretation of queries from users andapplications based on the existing or updated metadata information aboutthose existing systems and entities. For the specifics of the presentdisclosure, the cloud system 102 and its respective cloud servers 104,138 can provide the processing power (i.e., processors 108, 140) andthreads to perform the metadata refreshing to ensure that accuratequeries and presentations of information are performed for users andapplications.

As illustrated, one or more cloud servers 104, 138 can be included inthe cloud system 102. Cloud system 102 may also be associated with or apart of network 150 in some instances. The processors 108, 140 of thecloud system 102 and its various servers 104, 138 can be shared to allowfor improved processing and processing power. While illustrated asservers, the components on the cloud system 102 may be non-serversystems, including individual processors on non-server computers.

As illustrated, example cloud server 104 includes interface 106, one ormore processors 108, a metadata refresh module 110, and memory 122. Theinterface 106 is used by the cloud server 104 for communicating withother systems in a distributed environment—including within theenvironment 100—connected to the cloud system 102 and/or network 150,e.g., the external data sources 160, client 180, other cloud servers104, 138, as well as other systems communicably coupled to the cloudsystem 102 and/or network 150. Generally, the interface 106 compriseslogic encoded in software and/or hardware in a suitable combination andoperable to communicate with the cloud system 102 and/or network 150.More specifically, the interface 106 may comprise software supportingone or more communication protocols associated with communications suchthat the cloud system 102, network 150, and/or interface's hardware isoperable to communicate physical signals within and outside of theillustrated environment 100. Still further, the interface 106 may allowthe cloud server 104 to communicate with the external data sources 160to access updated information related to the database model metadata 172associated with particular metadata modules 124 as described in thepresent disclosure.

Network 150 facilitates wireless or wireline communications between thecomponents of the environment 100 (e.g., between the cloud servers 104and the external data sources 160), as well as with any other local orremote computer, such as additional mobile devices, clients (e.g.,client 180), servers, or other devices communicably coupled to network150, including those not illustrated in FIG. 1. In the illustratedenvironment, the network 150 is depicted as a single network, but may becomprised of more than one network without departing from the scope ofthis disclosure, so long as at least a portion of the network 150 mayfacilitate communications between senders and recipients. In someinstances, one or more of the illustrated components (e.g., the cloudsystem 102) may be included within network 150 as one or morecloud-based services or operations. The network 150 may be all or aportion of an enterprise or secured network, while in another instance,at least a portion of the network 150 may represent a connection to theInternet. In some instances, a portion of the network 150 may be avirtual private network (VPN). Further, all or a portion of the network150 can comprise either a wireline or wireless link. Example wirelesslinks may include 802.11a/b/g/n/ac, 802.20, WiMax, LTE, and/or any otherappropriate wireless link. In other words, the network 150 encompassesany internal or external network, networks, sub-network, or combinationthereof operable to facilitate communications between various computingcomponents inside and outside the illustrated environment 100. Thenetwork 150 may communicate, for example, Internet Protocol (IP)packets, Frame Relay frames, Asynchronous Transfer Mode (ATM) cells,voice, video, data, and other suitable information between networkaddresses. The network 150 may also include one or more local areanetworks (LANs), radio access networks (RANs), metropolitan areanetworks (MANs), wide area networks (WANs), all or a portion of theInternet, and/or any other communication system or systems at one ormore locations.

The cloud server 104 also includes one or more processors 108. Althoughillustrated as multiple processors 108 in FIG. 1, a single processor maybe used according to particular needs, desires, or particularimplementations of the environment 100. Each processor 108 may be acentral processing unit (CPU), an application specific integratedcircuit (ASIC), a field-programmable gate array (FPGA), or anothersuitable component. Generally, the processor 108 executes instructionsand manipulates data to perform the operations of the cloud server 104,in particular those related to refreshing metadata modules 124.Specifically, the processors 108 execute the algorithms and operationsdescribed in the illustrated figures, as well as the various softwaremodules and functionality, including the functionality for sendingcommunications to and receiving transmissions from clients 180 andexternal data sources 160, as well as to other devices and systems. Eachprocessor 108 may have a single or multiple core, with each coreavailable to host and execute an individual processing thread. Toperform the concurrent refreshing as described herein, multiple coresare available within the cloud server 104 and/or the cloud system 102and its other servers 138 and/or processors available.

Regardless of the particular implementation, “software” includescomputer-readable instructions, firmware, wired and/or programmedhardware, or any combination thereof on a tangible medium (transitory ornon-transitory, as appropriate) operable when executed to perform atleast the processes and operations described herein. In fact, eachsoftware component may be fully or partially written or described in anyappropriate computer language including C, C++, JavaScript, Java™,Visual Basic, assembler, Perl®, any suitable version of 4GL, as well asothers.

Using the processors 108, the cloud server 104 can execute variousapplications and processes, including those illustrated in FIG. 1. Whilethe cloud server 104 may be capable of performing various other tasks,the present disclosure is directed to the cloud server's refreshing ofmetadata relating to one or more data sources or entities to whichqueries are executed from, either by the cloud server 104 or a relatedsystem that uses the metadata refreshed by the cloud server 104. Assuch, a metadata refresh module 110 is executed by the cloud server 104to perform the functionality associated with the refresh process.

The metadata refresh module 110 is used to identify and update metadatadefined in a plurality of metadata modules 124, such as those stored inmemory 122. Metadata modules 124 can be single or groups of metadatadefining or associated with one or more entities. Each metadata modules124 can include an identifier (ID) 126 identifying the correspondingentity or data structure to which it refers. Individual entities maydescribe particular target objects. Metadata modules 124 may combinerelated entities into a single metadata file or module. For example, amodule called “Employee_Simple” may include entities such asPersonal_Information, Job_Entity, Employment_Information, Home_Address,and other related entities. As that information is closely related,modulating the entities allows for a close connection between theinformation that may be requested or used in related searches or otheroperations. Additional information about the particular entity or datastructure may be provided to allow users and/or other applications toclearly identify the corresponding data source or data set, such as aparticular address or location of the metadata to be used in refreshingthe data. Further, the metadata modules 124 are associated with a set ofmetadata information 128 that defines the data model structure of theunderlying data set, allowing queries associated with the underlyingdata structure or set to accurately match the current data model andaccess the full, current data set. When metadata for the particular datasource or data set is incorrect, queries and interactions to the backendsource may miss data, request non-existent data, or otherwise provideerroneous or incomplete query results. In some instances, metadatamodules 124 may be associated with a particular explicit priorityindication 130. In prioritizing refresh actions, this explicit priorityindication 130 may be used to treat a particular metadata module 124with a relatively higher priority than would normally be provided asdescribed below. Not all metadata modules 124 may be associated with anexplicit priority indication 130, and such indications may be based on auser- or administrator-level selection or flagging of particularmetadata modules 124 as of higher priority as compared to other modules.In some instances, different levels of relative priority may beidentified to allow for distinction between priority types or levels ofparticular modules 124. In some instances, the priority indication 130may be determined based on rules or dynamic determinations that identifya particular metadata module 124 as a higher priority module incomparison to other modules. Still further, some metadata modules 124associated with mission-critical or other high important data sets may,by default, be initially assigned a priority indication. Any suchautomatic indications may be modified at a later time by a user oradministrator as appropriate.

As illustrated, the metadata refresh module 110 includes a plurality ofsubmodules or agents, including a prioritization module 112, a systemstatus analyzer 114, a refresh type determination module 116, and arefresh handler 118. The various illustrated submodules may becomponents within the metadata refresh module 110, or may be integral tothe module 110 and simply part of its functionality.

The prioritization module 112 can perform analyses of particularmetadata modules 124 to be refreshed and assist in the prioritization oftheir refresh within a refresh queue or ordering system. In general,modules 124 may be prioritized according to their respective refreshexecution time. The longer the execution time of a particular metadatamodule 124, the higher its relative priority. By doing so, the wholeexecution time may be made as short as possible. For example, formodules M₁ and M₂, execution times for the refresh may be T₁ and T₂,respectively, where T₁>T₂. Since T₁>T₂, then the priority of module M₁(P₁) is relatively higher than the priority of M₂ (P₂), or P₁>P₂. Theprioritization module 112 or another suitable component may monitor andstore information on the relative refresh times. In such instances,initial refresh times within a system may be set to 0. Using a warm-upperiod for the refreshing, the average refresh time over a set number,n, of iterations can provide metric information related to the refreshtimes, which can be store in the metadata refresh metrics 132 of memory122. These average refresh times and/or at least some of the historicalcalculations can be maintained in memory 122 or elsewhere and can beused to determine the refresh time-based priority. The prioritizationmodule 112 can use that prior to each refresh to generate aprioritization order for the particular metadata modules 124 beingrefreshed. As noted above, at least some of the metadata modules 124 maybe associated with an explicit priority indication 130. In suchinstances, the modules 124 associated with the priority indication 130can be prioritized above those with relatively slower refresh times,where appropriate. Once the prioritization is complete, a currentprioritization list 136 can be generated, made available, and/orotherwise used to manage the order of the refresh. The currentprioritization list 136 can be a serialized list of refresh tasks to beperformed based on the prioritization information, where, in aconcurrent refresh system, two or more of the tasks may be performed ina concurrent refresh process.

A system status analyzer 114 of the metadata refresh module 110 canperform operations to determine the current status of the system, cloudserver(s) 104, and the cloud system 102. The analyzer 114 can performAPI calls and system analyzes to determine the dynamic status of thesystem resources at a particular time prior to refresh. In someinstances, a set of configurable refresh rules 134 may be available thatcan determine requirements of the system to allow a concurrent refreshto occur. Those requirements may include particular thresholds withinwhich the system must be in order to perform the concurrent refresh,including a current CPU usage, a current available CPU core amount, anumber of available CPU cores, an available memory amount, a number ofavailable database connections (“DB connections”), or any other suitableand relevant data. Such live information can be used at the time of arefresh workflow process to determine if and how to proceed. In someinstances, the system status analyzer 114 may access one or moreexternal systems or tools to identify the system's status, including forexample a cloud status monitor 142. The cloud status monitor 142 may beany tool or agent that can be used to measure and return at least aportion of the status information for a cloud or other server system.For example, the cloud status monitor 142 may include or be associatedwith one or more system status APIs 144 capable of accessing and/orcalculating information on current loads and usages on the system and/orindividual systems, services, or components thereof

A refresh type determination module 116 can be used to determine, basedon the output of, e.g., the system status analyzer, (1) whether a singlethread refresh or a concurrent thread refresh is to be used and, (2) ifthe concurrent thread refresh is to be used, the specifics of theparticular concurrent refresh. The refresh type determination module 116can perform checks, based on the configurable refresh rules 134, todetermine whether the system conditions and status allows for aconcurrent refresh. For example, if the configurable rules require lessthan 90% CPU usage and available memory for a refresh being higher thanthe required memory for the refresh, the refresh type determinationmodule 116 can identify whether all required parameters for a concurrentrefresh are met. If not, the refresh type determination module 116 canperform a single thread refresh, where the modules are sequentiallyrefreshed based on the current prioritization list 136. If, however, therequirements are met, then the refresh type determination module 116 candetermine the parameters of the particular concurrent refresh. Inparticular, the number of CPU cores and the idle DB connectionsavailable in the system or at the cloud server 104 are identified basedon information collected from the prior system analysis. Additionally,more than one idle DB connection must be available to allow for aconcurrent refresh. If the number of idle DB connections available,#conn, is greater than the number of CPU core number of the server, C,then a concurrent refresh session with C+1 threads can be executed toconcurrently run the metadata refresh. If #conn is less than or equal toC, but #conn is greater than 1, then a concurrent refresh session with#conn threads is to be executed.

The information about the type of refresh can, in some instances, beprovided to a refresh handler 118 for performing the refresh operations.The refresh handler 118 can initialize the determined type of refresh(i.e., single thread or concurrent thread) by reserving the resourcesrequired to perform the action (i.e., particular CPU cores, DBconnections, etc.). The refresh handler 118 may include or be associatedwith a DB connection manager 120 that can be used to allow the DBconnections to identify and connect to the appropriate external datasources 160 (or, in some instances, internal data sources) for themetadata refresh. For single thread refreshing, the refresh handler 118can cause a sequential refresh of the metadata modules 124 in the orderas defined in the current prioritization list 136. For concurrent threadrefreshing, a general queue may be formed in the order of the currentprioritization list 136. The refresh handler 118 can then add individualmetadata modules 124 into particular threads from the queue based onthat order. When a thread opens, or in some cases, while the thread isprocessing, another metadata module 124 from the list 136 can beprovided to the threads for processing.

Memory 122 of the cloud server 104 may represent a single memory ormultiple memories. The memory 122 may include any memory or databasemodule and may take the form of volatile or non-volatile memoryincluding, without limitation, magnetic media, optical media, randomaccess memory (RAM), read-only memory (ROM), removable media, or anyother suitable local or remote memory component. The memory 122 maystore various objects or data (e.g., the metadata modules 124, metadatarefresh metrics 132, configurable refresh rules 134, currentprioritization list 136, etc.), including financial data, userinformation, administrative settings, password information, caches,applications, backup data, repositories storing business and/or dynamicinformation, and any other appropriate information associated with thecloud server 104 including any parameters, variables, algorithms,instructions, rules, constraints, or references thereto. Additionally,the memory 122 may store any other appropriate data, such as VPNapplications, firmware logs and policies, firewall policies, a securityor access log, print or other reporting files, as well as others.

As illustrated and previously mentioned, one or more additional cloudservers 138 may be available to host threads associated with themetadata refresh where one or more of their CPU cores associated withtheir one or more processors 140 are available at the time of therefresh. As the cloud system 102 allows for distributed processing, aload balancer (not shown) may be used to manage the load provided to anyparticular system, processor 108, 140, or cores of those processors 108,140.

The external data sources 160 illustrated in FIG. 1 may be any suitabledata source, including Web services, databases, spreadsheets,applications with a defined metadata schema, or any other suitablesource. The metadata modules 124 may be associated with particular onesof the external data sources 160 (or various parts of the same externaldata sources 160 where such modulation is available). The external datasources 160 may be a full system or server, a particular program orresource within an external system from the cloud system 102, adatabase, or a component or data source located within or part of thecloud system 102.

Each external data source 160 may be associated with an application 166,but is not required to do so. The application 166 may be a Web serviceor other applications capable of interacting with the underlyingdatabase 170, e.g., where queries may be received and processed withinthe external data source 160 and executed on the database 170. Database170 may be a relational database, in-memory columnar database,spreadsheet, or any other source with a defined data model. In someinstances, memory 168 of the external data source 160 may include adatabase 170 and a set of data model metadata 172 defining the structureof the database 170. In some instances, the metadata 172 may beexplicitly defined, while in others the metadata 172 may be implicitlyidentified based on an analysis of the database 170. Memory 168 may besimilar to or different from memory 122. Communication with the externaldata source 160 may be via an interface 162, which may be similar to ordifferent from interface 106. The application 166 and/or otherfunctionality of the data source 160 may be executed by the processor(s)164, which may be similar to or different from processor 108. In someinstances, the application 166 may include a monitor 167, which may bean agent or other module of the application 166 or of the metadatarefresh module 110 which can determine when particular changes to anunderlying database 170 and its associated data model structure occurwithin the external data source 160. In response to determining a changehas occurred that may affect the data model and the associated metadata,the monitor 167 may report or send a notification to the metadatarefresh module 110, where a refresh for at least the associated externaldata source's 160 associated metadata module 124 is triggered.

As illustrated, one or more clients 180 may be present in the examplesystem 100. Each client 180 may be associated with requests received bythe cloud system 102 related to the one or more data sources 160 andwhich request data associated with those data sources, where thoserequests and queries require knowledge and current information relatedto the metadata of each data source 160. As illustrated, the client 180may include an interface 182 for communication (similar to or differentfrom interface 106), a processor 184 (similar to or different fromprocessor 108), a client application 186, memory 188 (similar to ordifferent from memory 122), and a graphical user interface (GUI) 190.

The illustrated client 180 is intended to encompass any computing devicesuch as a desktop computer, laptop/notebook computer, mobile device,smartphone, personal data assistant (PDA), tablet computing device, oneor more processors within these devices, or any other suitableprocessing device. In general, the client 180 and its components may beadapted to execute any operating system, including Linux, UNIX, Windows,Mac OS®, Java™, Android™, or iOS. In some instances, the client 180 maycomprise a computer that includes an input device, such as a keypad,touch screen, or other device(s) that can interact with the clientapplication 186, and an output device that conveys informationassociated with the operation of the applications and their applicationwindows to the user of the client 180. Such information may includedigital data, visual information, or a GUI 190, as shown with respect tothe client 180. Specifically, the client 180 may be any computing deviceoperable to communicate queries or communications to the cloud system102, external data sources 160, other clients 180, and/or othercomponents via network 150, as well as with the network 150 itself,using a wireline or wireless connection. In general, client 180comprises an electronic computer device operable to receive, transmit,process, and store any appropriate data associated with the environment100 of FIG. 1.

GUI 190 of the client 180 interfaces with at least a portion of theenvironment 100 for any suitable purpose, including generating a visualrepresentation of the client application 186. In particular, the GUI 190may be used to present results of queries associated with one or moreexternal data sources 160. GUI 190 may also be used to view and interactwith various Web pages, applications, and Web services located local orexternal to the client 180. Generally, the GUI 190 provides the userwith an efficient and user-friendly presentation of data provided by orcommunicated within the system. The GUI 190 may comprise a plurality ofcustomizable frames or views having interactive fields, pull-down lists,and buttons operated by the user. For example, the GUI 190 may provideinteractive elements that allow a user to view or interact withinformation related to the operations of processes associated with thecloud system 102. In general, the GUI 190 is often configurable,supports a combination of tables and graphs (bar, line, pie, statusdials, etc.), and is able to build real-time portals, applicationwindows, and presentations. Therefore, the GUI 190 contemplates anysuitable graphical user interface, such as a combination of a genericweb browser, a web-enable application, intelligent engine, and commandline interface (CLI) that processes information in the platform andefficiently presents the results to the user visually.

While portions of the elements illustrated in FIG. 1 are shown asindividual modules that implement the various features and functionalitythrough various objects, methods, or other processes, the software mayinstead include a number of sub-modules, third-party services,components, libraries, and such, as appropriate. Conversely, thefeatures and functionality of various components can be combined intosingle components as appropriate.

FIG. 2 represents an example flow for concurrently refreshing metadataassociated with a plurality of modules. For clarity of presentation, thedescription that follows generally describes method 200 in the contextof the system 100 illustrated in FIG. 1. However, it will be understoodthat method 200 may be performed, for example, by any other suitablesystem, environment, software, and hardware, or a combination ofsystems, environments, software, and hardware as appropriate.

At 205, a triggering of a metadata refresh occurs and is identified by acloud server or a suitable system managing the refresh of the associatedmetadata. The triggering may occur in response to a particular event orinput, an explicit request from a user or administrator, an expiredcache storing the metadata, a determination that one or more entities,parameters, or properties associated with a particular data source havechanged, a programmed update included in executing code associated withan application, or any other suitable trigger. In some instances, only asubset of the metadata modules may be refreshed, such as when changes tobackend databases cause only a subset of modifications and refreshes tobe needed, or where only specific sets of metadata are in need ofrefreshing.

At 210, which may occur concurrently with 205, prior to 205, or at anyother suitable time, a metadata modulation process may occur. Themetadata modulation process may associate one or more sets of metadataassociated with related data sources or entities, such that refreshingof particular metadata modules may refresh related metadata sets (e.g.,at a single location, used for a single application or purpose, etc.).The metadata sets can be grouped into small modules according todatabase relationships and business logic, as needed. Additionally, newentities may be registered upon creation such that new modules arecreated or existing modules are updated to include the newly registeredentities or data sources.

At 215, the set of metadata modules to be refreshed are prioritized. Asdescribed previously, metadata modules may be previously calculated tohave an average refresh execution time based on prior refreshing. Forgeneral prioritization, the modules can be ordered in a descending orderbased on the length of the average refresh time. In doing so, theslowest metadata modules can be handled initially while the potentiallyconcurrent process allows faster modules to be processed concurrently tothe relatively slower modules. Further, some metadata modules, based ontheir business value and/or criticality, may be explicitly identified ashigh priority modules. In such instances, those explicitly higherpriority modules can be prioritized first before considerations ofrelative refresh times are considered. Alternatively, a weightedanalysis may be used, where appropriate. Other means of prioritizing themetadata modules to be refreshed can also be used.

At 220, a set of requirements within the system for performing aconcurrent metadata refresh process may be identified. The requirementsmay be hardcoded requirements or they may be user- oradmin-configurable. Example requirements may include a particularpercentage of current CPU usage, an absolute or relative amount ofmemory available for the process, as well as any other suitableparameters. In some instances, the requirements may be dynamic, or mayshift based on recent or current conditions, as well as on recenthistorical system status information. At 225, a determination is made asto whether the current system status exceeds the concurrent refreshrequirements. The current system status may be determined by a systemmonitor, calls to one or more status-related APIs which return statusinformation, or any other suitable determination. If a determination ismade that the requirements are not met, method 200 continues at 235,where a single thread refresh is executed using a single thread tosequentially perform the refresh process. If, however, the requirementsare satisfied, method 200 continues at 230. At 230 a determination ismade as to whether more than one (1) idle database connection isavailable (or another number greater than one as defined by theconcurrent refresh requirements). If only one database connection isavailable, only one metadata module can be refreshed at a time such thatmethod 200 continues at 235 to perform the single thread refresh. If,however, more than one idle database connection is available, method 200continues to 240.

At 240, a determination is made regarding the relative number of idledatabase connections, or #conn, and the number of available CPU cores,C, which determines the number of possible threads for processing. Ifthe #conn is greater than C, then method 200 continues at 245 where aconcurrent refresh using C+1 threads is executed. In such a situation,the first C+1 modules in the prioritized list are each sent to thethreads for refreshing. As a particular module is completed, a newmodule is sent to the available thread for processing. In someinstances, one or more of the modules may be assigned to a particularthread prior to the previous module's update being completed. If,however, #conn is less than or equal to C, method 200 continues at 250where a concurrent refresh using a number of threads equal to #conn, thenumber of idle database connections, is executed.

Upon completion of the refreshing, method 200 can wait for a newtriggering action or event to occur, whereby the process occurs againbased on the particular set of metadata modules being refreshed.

FIG. 3 is an example illustration 300 of a workflow for concurrentthread metadata refresh where the idle number of database connections,#conn, is greater than the number of CPU cores, C, in one or moresystems performing the refreshing. As noted, during the determination ofthe thread numbers, the number of connections is greater than the numberof CPU cores for refresh processing. In such instances, the number ofthreads is C+1. For example, if two (2) CPU cores are available, three(3) threads will be created, shown as thread 310, 315, and 320. A queue305 stores the prioritized set of metadata modules to be refreshed. Uponinstantiated the threads, those metadata modules are provided toparticular threads for execution in a concurrent manner. As particularrefreshes are completed, new modules are provided to the now availablethread.

FIG. 4 is an example illustration 400 of a workflow for concurrentthread metadata refresh where the idle number of database connections,#conn, is less than or equal to the number of CPU cores, C, in the oneor more systems performing the refreshing. Because the system is limitedby the number of connections that are available for refreshing themodules, the number of threads are set to the highest possible number ofdatabase connections, #conn. Similar to FIG. 3, a queue 405 ofprioritized metadata modules is created. A number of threads, such asexample threads 410, 415, and 420, numbering equal to #conn areinstantiated, with the relatively highest priority metadata modulesbeing initially provided to those example threads 410, 415, and 420.

The preceding figures and accompanying description illustrate exampleprocesses and computer implementable techniques. But environment 100 (orits software or other components) contemplates using, implementing, orexecuting any suitable technique for performing these and other tasks.It will be understood that these processes are for illustration purposesonly and that the described or similar techniques may be performed atany appropriate time, including concurrently, individually, or incombination. In addition, many of the steps in these processes may takeplace simultaneously, concurrently, and/or in different orders than asshown. Moreover, environment 100 may use processes with additionalsteps, fewer steps, and/or different steps, so long as the methodsremain appropriate.

In other words, although this disclosure has been described in terms ofcertain embodiments and generally associated methods, alterations andpermutations of these embodiments and methods will be apparent to thoseskilled in the art. Accordingly, the above description of exampleembodiments does not define or constrain this disclosure. Other changes,substitutions, and alterations are also possible without departing fromthe spirit and scope of this disclosure.

What is claimed is:
 1. A computer-implemented method performed by atleast one processor executing instructions, the method comprising:identifying a set of metadata modules associated with one or moreentities, the metadata modules defining metadata associated with aparticular data model for the associated entities; identifying a requestto initiate a refreshing of the metadata for a subset of the set ofmetadata modules; prioritizing each metadata module from the subset ofthe set of metadata modules into a prioritization order; determiningwhether two or more idle database connections are available; and inresponse to determining that two or more idle database connections areavailable, initiating a concurrent refresh of the subset of the set ofmetadata modules in the prioritization order.
 2. The method of claim 1,further comprising, in response to determining that only one idledatabase connection is available, initiating a single thread refresh ofthe subset of the set of metadata modules.
 3. The method of claim 2,wherein the single thread refresh of the set of metadata modulescomprises a sequential processing of metadata modules in the subset ofthe set of metadata modules.
 4. The method of claim 1, furthercomprising, prior to determining whether two or more idle databaseconnections are available: determining whether a set of system resourcemetrics satisfy a concurrent refresh threshold; and in response todetermining the set of system resource metrics do not satisfy theconcurrent refresh threshold, initiating a single thread refresh of thesubset of the set of metadata modules.
 5. The method of claim 4, whereinthe concurrent refresh threshold includes at least one of a particularcurrent CPU usage amount threshold and a current available memorythreshold, wherein the current available memory threshold represents anavailable memory greater than the required memory associated with theconcurrent refresh.
 6. The method of claim 1, wherein initiating theconcurrent refresh of the subset of the set of metadata modulescomprises: determining a CPU core number, C, of a particular set ofservers to perform the refresh; determining the number of idle databaseconnections, #CONN, at the particular set of servers; in response todetermining that C is less than or equal to #CONN, performing aconcurrent refresh of the subset of the set of metadata modules using#CONN threads at the particular set of servers; and in response todetermining that C is greater than #CONN, performing a concurrentrefresh of the subset of the set of metadata modules using C+1 threadsat the particular set of servers.
 7. The method of claim 1, wherein therequest to initiate a refreshing of the metadata for a subset of the setof metadata modules comprises one of: a manual request by a particularuser to initiate the refreshing of the metadata; an automatic request toinitiate the refreshing of the metadata in response to the expiration ofa cache associated with the metadata modules; an automatic request toinitiate the refreshing of the metadata in response to a determinationthat at least one entity associated with at least one of the metadatamodules has been updated; and an automatic request to initiate therefreshing of the metadata in response to a determination that at leastone new entity is newly added.
 8. The method of claim 1, whereinprioritizing each metadata module from the subset of the set of metadatamodules into a prioritization order comprises prioritizing the metadatamodules in a descending order based on an average refresh time for eachof the metadata modules.
 9. The method of claim 8, wherein at least onemetadata module in the subset of the set of metadata modules isassociated with a relatively higher priority, the relatively higherpriority distinct from the average refresh time of the at least onemetadata module, wherein the at least one metadata module is provided arelatively higher priority in the prioritization order than metadatamodules having a relatively longer average refresh time than the atleast one metadata module.
 10. The method of claim 8, wherein theaverage refresh time for each metadata module is calculated based onmetrics calculated for a predetermined number of prior refresh times foreach metadata module.
 11. A system comprising: at least one processor;and a memory communicatively coupled to the at least one processor, thememory storing instructions which, when executed, cause the at least oneprocessor to perform operations comprising: identifying a set ofmetadata modules associated with one or more entities, the metadatamodules defining metadata associated with a particular data model forthe associated entities; identifying a request to initiate a refreshingof the metadata for a subset of the set of metadata modules;prioritizing each metadata module from the subset of the set of metadatamodules into a prioritization order; determining whether two or moreidle database connections are available; and in response to determiningthat two or more idle database connections are available, initiating aconcurrent refresh of the subset of the set of metadata modules in theprioritization order.
 12. The system of claim 11, the operations furthercomprising, in response to determining that only one idle databaseconnection is available, initiating a single thread refresh of thesubset of the set of metadata modules.
 13. The system of claim 12,wherein the single thread refresh of the set of metadata modulescomprises a sequential processing of metadata modules in the subset ofthe set of metadata modules.
 14. The system of claim 11, the operationsfurther comprising, prior to determining whether two or more idledatabase connections are available: determining whether a set of systemresource metrics satisfy a concurrent refresh threshold, wherein theconcurrent refresh threshold includes at least one of a particularcurrent CPU usage amount threshold and a current available memorythreshold, wherein the current available memory threshold represents anavailable memory greater than the required memory associated with theconcurrent refresh; and in response to determining the set of systemresource metrics do not satisfy the concurrent refresh threshold,initiating a single thread refresh of the subset of the set of metadatamodules.
 15. The system of claim 11, wherein initiating the concurrentrefresh of the subset of the set of metadata modules comprises:determining a CPU core number, C, of a particular set of servers toperform the refresh; determining the number of idle databaseconnections, #CONN, at the particular set of servers; in response todetermining that C is less than or equal to #CONN, performing aconcurrent refresh of the subset of the set of metadata modules using#CONN threads at the particular set of servers; and in response todetermining that C is greater than #CONN, performing a concurrentrefresh of the subset of the set of metadata modules using C+1 threadsat the particular set of servers.
 16. The system of claim 11, whereinthe request to initiate a refreshing of the metadata for a subset of theset of metadata modules comprises one of: a manual request by aparticular user to initiate the refreshing of the metadata; an automaticrequest to initiate the refreshing of the metadata in response to theexpiration of a cache associated with the metadata modules; an automaticrequest to initiate the refreshing of the metadata in response to adetermination that at least one entity associated with at least one ofthe metadata modules has been updated; and an automatic request toinitiate the refreshing of the metadata in response to a determinationthat at least one new entity is newly added.
 17. The system of claim 11,wherein prioritizing each metadata module from the subset of the set ofmetadata modules into a prioritization order comprises prioritizing themetadata modules in a descending order based on an average refresh timefor each of the metadata modules, wherein at least one metadata modulein the subset of the set of metadata modules is associated with arelatively higher priority, the relatively higher priority distinct fromthe average refresh time of the at least one metadata module, whereinthe at least one metadata module is provided a relatively higherpriority in the prioritization order than metadata modules having arelatively longer average refresh time than the at least one metadatamodule.
 18. A non-transitory computer-readable medium storinginstructions which, when executed, cause at least one processor toperform operations comprising: identifying a set of metadata modulesassociated with one or more entities, the metadata modules definingmetadata associated with a particular data model for the associatedentities; identifying a request to initiate a refreshing of the metadatafor a subset of the set of metadata modules; prioritizing each metadatamodule from the subset of the set of metadata modules into aprioritization order; determining whether two or more idle databaseconnections are available; and in response to determining that two ormore idle database connections are available, initiating a concurrentrefresh of the subset of the set of metadata modules in theprioritization order.
 19. The medium of claim 18, the operations furthercomprising, in response to determining that only one idle databaseconnection is available, initiating a single thread refresh of thesubset of the set of metadata modules, wherein the single thread refreshof the set of metadata modules comprises a sequential processing ofmetadata modules in the subset of the set of metadata modules.
 20. Themedium of claim 18, the operations further comprising, prior todetermining whether two or more idle database connections are available:determining whether a set of system resource metrics satisfy aconcurrent refresh threshold, wherein the concurrent refresh thresholdincludes at least one of a particular current CPU usage amount thresholdand a current available memory threshold, wherein the current availablememory threshold represents an available memory greater than therequired memory associated with the concurrent refresh; and in responseto determining the set of system resource metrics do not satisfy theconcurrent refresh threshold, initiating a single thread refresh of thesubset of the set of metadata modules.